Playground PR for verification caching exploration#3457
Playground PR for verification caching exploration#3457
Conversation
| r.CopyVerificationInfo(cacheHit) | ||
| } else { | ||
| isEverythingCached = false | ||
| break |
There was a problem hiding this comment.
I was playing around converting this to slices.ContainsFunc, but then I realized maybe it's not optimal to break here? If we run into something that isn't yet cached, everything after it won't be updated from cache. Maybe that's fine though?
(also dunno what I think about the Func functions in slices; I guess the main use case is to not dupe logic in loops, but that doesn't apply here)
There was a problem hiding this comment.
Yeah, this is just kinda unfortunate. If we find something that's not cached, we have to rerun FromData, which will return all the results for this chunk - whether they're cached or not. So there's no reason to read from the cache. (Did I successfully explain that?)
There was a problem hiding this comment.
So there's no reason to read from the cache. (Did I successfully explain that?)
Essentially, this is a stopgap solution to handle duplicate chunks (e.g., scanning orgs + members that have multiple forks of kubernetes/kubernetes.)
Proper caching on a granular level will require splitting FromData and Verify into discrete functions.
// pseudo-code
cache := cache.New()
for _, chunk := range chunks {
for _, d := range detectors {
results := d.FromData(ctx, chunk)
for result := range results {
if val, ok := cache.Get(detector.Type, result.Raw); ok {
result.Update(val)
} else {
detector.Verify(result)
cache.Set(detector.Type, result.Raw)
}
}
}
}Result.DecoderType is only ever used by ResultWithMetadata (via its embedded Result). This unnecessarily complicates the relationship between the types and adds some warts to #3457, so this PR moves DecoderType directly into the only struct which actually uses it.
|
Actually implemented by #3801 |
Description:
This is a playground branch I made to play around with some ideas for introducing verification caching cleanly. Please don't merge it! Ever!
Checklist:
make test-community)?make lintthis requires golangci-lint)?